End-to-End Neural Video Coding Using a Compound Spatiotemporal Representation

Recent years have witnessed rapid advances in learnt video coding. Most algorithms solely relied on the vector-based motion representation and resampling (e.g., optical flow based bilinear sampling) for exploiting inter frame redundancy. In spite of great success adaptive kernel-based convolutions deformable convolutions) prediction uncompressed videos, integrating such approaches with rate-distortion optimization coding has been less successful. Recognizing that each solution offers unique advantages regions different texture characteristics, we propose a hybrid compensation (HMC) method adaptively combines predictions generated by these two approaches. Specifically, generate compound spatiotemporal (CSTR) through recurrent information aggregation (RIA) module using from current multiple past frames. We further design one-to-many decoder pipeline to CSTR, including resampling, mode selection maps enhancements, them achieve more accurate prediction. Experiments show our proposed system can provide better motion-compensated is robust occlusions complex motions. Together jointly trained intra coder residual coder, overall yields state-of-the-art efficiency low-delay scenario, compared traditional H.264/AVC H.265/HEVC, as well recently published learning-based methods, terms both PSNR MS-SSIM metrics.